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(54) IMAGE PICKUP DEVICE AND ITS PICTURE PROCESSING METHOD 

(57)Abstract: 

PURPOSE: To obtain a digital picture with excellent 
S/N, high gradation, large dynamic range and high 
picture quality by the digital image pickup device. 
CONSTITUTION: A same object is picked up 
simultaneously or continuously by changing exposure 
and resulting picture data are converted into digital data, 
which are stored in a memory. Then the fluctuation of 
picture data are corrected to implement picture 
matching processing (step 11), the gain of the picture 
data is adjusted to make the levels coincident with each 
other (step 1 2). Then the picture data are added to 
synthesize the pictures (step 13), the resulting picture is 
converted into R, G, B signals by process processing of 
a next stage (step 14) and outputted to the monitor as a 
luminance signal Y and a color difference signal C (step 
15). 
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* NOTICES * 

JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 

CLAIMS 
[Claim(s)] 

[Claim 1] Image pick-up equipment characterized by having a conversion means to digitize two 

or more image data of the same photographic subject which changed and picturized light 

exposure, an agreement processing means to perform alignment of the image of each digitized 

image data, a level adjustment means to double the level of each digitized image data, and an 

image composition means to compound the image data which doubled level. 

[Claim 2] A level adjustment means is image pick-up equipment according to claim 1 

characterized by comparing the average value of the part with which image data is saturated, 

and the field which excepted the circumference, and doubling level. 

[Claim 3] A level adjustment means is image pick-up equipment according to claim 1 

characterized by extracting a field with little brightness change of image data, comparing the 

average value of the field, and doubling level. 

[Claim 4] A level adjustment means is image pick-up equipment according to claim 1 
characterized by extracting the field near the white of image data, comparing the average value 
of the field, and doubling level. 

[Claim 5] A level adjustment means is image pick-up equipment according to claim 1 
characterized by comparing the average value of the field which excepted the part with which 
the field of the arbitration of image data was extracted and the field is saturated, and doubling 
level. 

[Claim 6] There is no claim 1 which uses as the data by the side of low brightness the image 
data obtained with more light exposure than a criterion, uses as the data by the side of high 
brightness the image data obtained with standard light exposure, and is characterized by 
compounding image data for the image data obtained with light exposure smaller than a criterion 
in the combination of all or arbitration as data by the side of super-high brightness, and an image 
composition means is image pick-up equipment of a publication 5 either. 
[Claim 7] An image composition means is image pick-up equipment according to claim 6 
characterized by compounding image data by making into a fixed value the data value from which 
the combination of image data switches. 

[Claim 8] An image composition means is image pick-up equipment according to claim 6 
characterized by giving a certain width of face from which the content ratio of the data by the 
side of high brightness and the data by the side of low brightness can change to the data value 
from which the combination of image data switches, and compounding image data to it. 
[Claim 9] An image composition means is image pick-up equipment according to claim 8 
characterized by changing linearly the content ratio of the data by the side of high brightness, 
and the data by the side of low brightness, and compounding an image. 
[Claim 10] An image composition means is image pick-up equipment according to claim 8 
characterized by changing the content ratio of the data by the side of high brightness, and the 
data by the side of low brightness in a cosine curve, and compounding an image. 
[Claim 1 1] The image-processing approach characterized by compounding those image data 
after making the level of each image data in agreement, while changing light exposure for the 
same photographic subject, changing into digital data coincidence or each image data which 
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picturized continuously and was obtained and performing alignment of an image to the digitized 
image data. 

[Translation done.] 
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* NOTICES * 

JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[Industrial Application] Especially this invention is a low noise and relates to image pick-up 
equipment with gradation able to obtain the large digital image of a dynamic range highly, and its 
image-processing approach. 
[0002] 

[Description of the Prior Art] Recently, the digital-type image pick-up equipment which deal with 
the picture signal which picturized the photographic subject as a digital signal, such as a digital 
video movie camera and a digital still video camera, is being put in practical use. With such image 
pick-up equipment, after sample hold of the output signal of a solid state image pickup device is 
carried out, the gain adjustment of it is carried out, and it is changed into a digital signal from an 
analog signal by the A/D-conversion circuit, and is recorded on memory. And after transform 
processing of the picture signal from this digital-signaHzed solid state image pickup device is 
carried out to the luminance signal and color-difference signal of NTSC/PAL system, or it is 
processed as R, G, and a B signal, looking it like [ a D/A conversion circuit ] and changing it into 
an analog signal more, it is outputted to a monitor or is inputted into a computer etc. as digital 
image information again. 

[0003] Here, although the gradation of the above digital image pick-up equipments is decided by 
the A/D-conversion circuit, the thing of 8 bits - 10 bits of current is put in practical use by this 
A/D-conversion circuit. However, it is inadequate for the gradation in the low brightness section 
of a photographic subject to this extent then, and becomes coarse image quality, and displeasure 
is given. 

[0004] Moreover, although the noise generated with a solid state image pickup device and the 
random noise generated by the analog circuit system to an A/D-conversion circuit input will 
determine S/N of the last image with conventional image pick-up equipment, this is also 
conspicuous in the low brightness section, and the good image of high S/N is not obtained. 
[0005] 

[Problem(s) to be Solved by the Invention] Since S/N of the last image was determined by the 
noise which conventional digital-type image pick-up equipment is constituted as mentioned 
above, and gradation is determined by the A/D-conversion circuit, and is generated with an 
image sensor, and the random noise generated in an analog circuit, there was a trouble that the 
image quality of the low brightness section deteriorated. 

[0006] Then, in this case, although it is possible to carry out a gain adjustment on an operation 
after enlarging light exposure, picturizing it and digital-signal-izing that image pick-up signal, 
since the dynamic range is low, the high brightness section will be saturated and the information 
on the high brightness section will be lost with conventional image pick-up equipment. 
[0007] This invention was made paying attention to the above troubles, and its S/N is good and 
it aims at offering the image pick-up equipment which can obtain the high-definition digital image 
with a large dynamic range which is high gradation, and its image-processing approach. 
[0008] 

[Means for Solving the Problem] The image pick-up equipment of this invention is constituted as 
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follows. 

[0009] (1) It had a conversion means to digitize two or more image data of the same 
photographic subject which changed and picturized light exposure, an agreement processing 
means to perform alignment of the image of each digitized image data, a level adjustment means 
to double the level of each digitized image data, and an image composition means to compound 
the image data which doubled level. 

[0010] (2) In the image pick-up equipment of the above (1), a level adjustment means compares 
the average value of the part with which image data is saturated, and the field which excepted 
the circumference, and doubled level. 

[0011] (3) In the image pick-up equipment of the above (1), a level adjustment means extracts a 
field with little brightness change of image data, compares the average value of the field, and 
doubled level. 

[0012] (4) In the image pick-up equipment of the above (1), a level adjustment means extracts 
the field near the white of image data, compares the average value of the field, and doubled level. 

[0013] (5) In the image pick-up equipment of the above (1), a level adjustment means compares 
the average value of the field which excepted the part with which the field of the arbitration of 
image data was extracted and the field is saturated, and doubled level. 

[0014] (6) The above (1) thru/or (5) In which image pick-up equipment, an image composition 
means uses as the data by the side of low brightness the image data obtained with many light 
exposure from the criterion, uses as the data by the side of high brightness the image data 
obtained with standard light exposure, and compounded image data for the image data obtained 
with light exposure smaller than a criterion in the combination of all or arbitration as data by the 
side of super-high brightness. 

[0015] (7) In the image pick-up equipment of the above (6), the image composition means 
compounded image data by making into a fixed value the data value from which the combination 
of image data switches. 

[0016] (8) In the image pick-up equipment of the above (6), an image composition means gives a 
certain width of face from which the content ratio of the data by the side of high brightness and 
the data by the side of low brightness can change to the data value from which the combination 
of image data switches, and compounded image data to it 

[0017] (9) In the image pick-up equipment of the above (8), an image composition means 
changes linearly the content ratio of the data by the side of high brightness, and the data by the 
side of low brightness, and compounded the image. 

[0018] (10) In the image pick-up equipment of the above (8), an image composition means 
changes the content ratio of the data by the side of high brightness, and the data by the side of 
low brightness in a cosine curve, and compounded the image. 

[0019] Moreover, after it makes the level of each image data in agreement while the image- 
processing approach of this invention changes light exposure for the same photographic subject, 
changes into digital data coincidence or each image data which picturized continuously and was 
obtained and performs alignment of an image to the digitized image data, it compounds those 
image data. 
[0020] 

[Function] The digital image which is obtained by photography of standard exposure, for example 
according to this invention, Processing of amending the blurring component of an image to two 
or more image data with the digital image obtained by coincidence or changing and picturizing 
light exposure continuously, One digital image with a large dynamic range is obtained with high 
S/N and high gradation by performing processing of the gain adjustment which makes in 
agreement the level of two or more digital images which differed in and obtained light exposure, 
and processing which compounds two or more of those digital image data. 
[0021] 

[Example] Drawing 1 is the block diagram showing the configuration of the image pick-up 
equipment by one example of this invention. In drawing, the solid state image pickup device 
which carries out photo electric translation of the image pick-up light from the same 
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photographic subject which la and 1b changed light exposure, and was picturized, an A-D 
(analogue to digital) conversion means by which 2 digitizes two or more image data from these 
image sensors 1a and 1b, and 3 amend a part for blurring of each digital image, and double a 
location with an agreement processing means to perform alignment of an image to each digitized 
image data here. 

[0022] An image processing predetermined in the picture signal which is the image composition 
means which compounds the image data which doubled level and is used as the data of one 
digital image, and is outputted from here in a non-illustrated process processing circuit is 
performed, serves as each chrominance signal of R, G, and B, a luminance signal of Y and C, and 
a color-difference signal, and a level adjustment means to double the level of each image data by 
which digitization of 4 was carried out [ above-mentioned ], and 5 are outputted. 
[0023] Next, actuation of the above-mentioned image pick-up equipment is explained. Drawing 2 
is a flow chart which shows the process of the image processing. 

[0024] First, from image sensors 1a and 1b, the image data (1) of the same photographic subject 
in the light exposure of two or more article affair and (2) are mostly incorporated in memory at 
the time of the same, when the same, here — the time of the same — ** — in the case of a 
camera with the optical system which combines the same photographic subject image with two 
or more solid state image pickup devices by prism etc. like a multi-plate type camera, it says, 
and is made by changing respectively the storage time of each solid state image pickup device, 
or putting an ND filter into the front face of a solid state image pickup device as a means the 
light exposure of each solid state image pickup device is made to differ in this case, moreover — 
almost — the time of the same — ** — saying — a veneer type camera — using — short 
spacing — and it is the case where the image incorporation by continuous shooting in different 
exposure conditions (it differs in the time of the amount of openings of drawing and a shutter 
second) is made. 

[0025] Reading appearance of two or more above-mentioned image data is carried out as an 
output of solid state image pickup devices 1a and lb, after sample hold is carried out, the gain 
adjustment of it is carried out and it is recorded on each memory after A-D conversion. 
[0026] Above-mentioned image data (1) and above-mentioned image data (2) are image data 
accumulated in memory after A-D conversion, and in this example, the data with which image 
data (1) was obtained by standard exposure, and image data (2) are data obtained by exposure 3 
times the light exposure of standard, and it is both digitized by the 10 bits (bit) A-D conversion 
circuit. Therefore, it has 1024 steps of gradation about. 

[0027] In addition, although 10 bits is not altogether used for image data actually and some 
offset parts are added, all explain 10 bits as a thing used as image information here. 
[0028] Image agreement processing is continuously performed by the above-mentioned image 
data (1) and image data (2) (step 11). This is not needed when the optical system of a multi- 
plate method is used as an incorporation means of an image, but since in the case of a veneer 
method image data (1) and image data (2) differ in time amount slightly and are photoed, the 
physical relationship of an image blurs slightly. Correcting a coordinate so that the data of image 
data (1) and image data (2) may be compared and the location of image data (1) and the location 
of image data (2) may be in agreement from this is performed. An approach like "blurring 
prevention" put in practical use with the current video movie camera etc. as the approach of 
this processing is adopted. 

[0029] Next, after the physical relationship of two images is put together, gain-adjustment 
processing is performed so that the data of image data (2) may be in agreement with the image 
level of image data (1) (step 12). Although what is necessary is just to double the image data (2) 
photoed with light exposure 3 times the light exposure of image data (1) 1/3 at this time, since it 
extracts and the light exposure of a camera has actually parts for an error, such as the error by 
the system, i.e., mechanism-errors, such as a shutter, the error which a photometry system has, 
a gain adjustment with a high precision by the comparison of actual image data is needed. 
[0030] Since a saturation part must except the data which compare the above, 1000 or less (it 
considers as a value to some extent lower than 1024 so that a saturation value may be certainly 
excepted without being thoroughly influenced by 1024 which is a saturation value of a noise etc.) 
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steps of image data (2), and image data (1) serve as level of the data which 333 or less steps 
compare, for example. 

[0031] Moreover, since the effect of a noise component becomes large, the data in the low 
brightness section need to except this. In this example, image data (1) makes it as 100 steps, 
and image data (2) makes the phase of a minimum 300 steps. Usually, since the level of 100% of 
whites is set as 1 / 3 - 1/4 of saturation, level to which it refers at a gain adjustment is 
performed in a part with the level of 30 - 40% or more of whites. 

[0032] And the average value of the field specified as mentioned above is taken in each image, B 
and the data value of each pixel of image data (2) are set [ the average value of image data (1) ] 
to C for the average value of A and image data (2), A/BxC is calculated and level of image data 
(2) is made the same with image data (1). 

[0033] Here, a gently-sloping field with less [ as other approaches of measuring the gain ratio of 
image data (1) and image data (2) ] change of the intensity level in a screen as an approach of 
acquiring high degree of accuracy than the above-mentioned approach can be extracted, and the 
approach of limiting only to a field with little the level change, and comparing the average of each 
image can be taken. 

[0034] If the field near the white in a screen may be extracted as an approach of further others, 
the average of only that field may be compared and it is made this approach, since the level 
balance of each color pixel (they are R, G, and B if the solid state image pickup device of the 
complementary color is used and it is Y, MG, G, Cy, and a pure color) is maintained, precision 
can be raised more. 

[0035] Moreover, by comparing the average in the decided field beforehand, unless it decides the 
comparison field beforehand like the middle-of-the-screen section or the middle~of-the-screen 
upper part and is saturated, although some precision falls, the processing time can be shortened 
substantially. 

[0036] After image agreement processing and gain-adjustment processing are completed as 
mentioned above, image addition processing in which image data (1) and image data (2) are added 
is performed (step 1 3). As for addition of this data, the data by the side of high brightness and 
image data (2) are used for image data (1) as data by the side of low brightness. 
[0037] Drawing 3 shows the above-mentioned addition art. Since about 1/(2) of image data is 3 
by previous gain-adjustment processing, the record level of image data (2) is about 341. It is the 
multiplier which the data of each image of image data (2) and the multiplier which N1 requires for 
the data of the pixel of image data (1), and N2 require for the data of the pixel of image data (2) 
at the data of each pixel of image data (1), and IN (2) of (b) at IN (1) of this graph (a). 
[0038] And according to the above-mentioned graph (a) and (b), (IN(1) xN1)+ (IN(2) xN2) is 
calculated, and it becomes one image. In this case, as for less than 330 data a little lower than 
the saturation value 341 of image data (2), the data of image data (2) are used, and, as for 330 or 
more data, the data of image data (1) are used. S/H of 1/3 or less part of the saturation of an 
image is set to one third, gradation becomes 3 times and the image of high definition and high 
information is built with doing in this way. 

[0039] However, by a part for the error of a gain adjustment etc., since the information on the 
data value 330 neighborhood is confused, the value before and behind a data value 330 appears 
as a false color, and possibility that the band of the likeness-color of a false etc. will arise 
remains here. 

[0040] Then, it is effective by (2) switching to image data (1) like drawing 4 and drawing 5 , giving 
the width of face of the data of a part, and changing the ratio gradually including image data (1) 
and the information on (2). Although drawing 4 changes the content ratio linearly and drawing 5 is 
changed in a cosine curve, it cannot be overemphasized that N1+N2 is always 1. By doing in this 
way, image data (1) and image data (2) can switch, the effect by the gain error in the section etc. 
can be reduced, and generating of a fake image can be suppressed. 

[0041] Next, by process processing (1) of the next step, transform processing of the image data 
compounded as mentioned above is carried out to R, G, and B signal, and it is outputted to them 
(step 14). Furthermore, it is changed into a luminance signal Y and a color-difference signal C by 
process processing (2) (step 15). And these signals can be inputted into a monitor TV, or are 
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inputted into a computer as image data. 

[0042] In addition, although image data (1) and image data (2) switched and ievei was brought to 
1 of saturation / place of four to 1 /about 3 in the above-mentioned example, the section which 
switches to the ****** quantity brightness section in that of the gamma curve attached by next 
process processing may be brought, and the effect of an above-mentioned gain error etc. can be 
reduced more in this case. 

[0043] And by performing the above image processings, it is good, and it is high gradation and 
S/N can obtain a high-definition digital image with a large dynamic range. 
[0044] Drawing 6 is a flow chart which shows actuation of other examples of this invention. 
Image data (1) and image data (2) are the same as the case of an above-mentioned example. 
Image data (3) is image data photoed by one third of exposure of standard light exposure. 
[0045] Alignment is made by image agreement processing (step 11), then, as for the three 
above-mentioned image data, the gain adjustment of three images is made (step 12). At this 
time, about 1/(2) of image data is set to 3, and it increases image data (3) 3 times. This expands 
the maximum of image data (3) to 3072. 

[0046] Next, image data (1) and image data (2) are compounded like the above-mentioned 
example, and image data (3) and image data (1) are compounded by the same method as 
composition of image data (1) and image data (2) (step 13). In this case, image data (1) and image 
data (3) switch, and level is made in 700 to 1000 steps of range. 

[0047] A dynamic range expands substantially the digital image outputted by performing such an 
image processing rather than the image outputted in the above-mentioned example. For this 
reason, even the photographic subject image of the super-high brightness section copies out 
vividly, and a low brightness image also has few noises and a clear image with high amount of 
information can be realized. 

[0048] In addition, composition of the above-mentioned image data (1) and image data (3) may 
be performed as other examples. In this case, mode of processing is the same as that of each 
above-mentioned example. Moreover, composition of image data (2) and image data (3) is 
sufficient. 

[0049] Moreover, in the above example, although explained by the case of 3 times of standard 
exposure, and 1/3 time, 4 times, 1/4 time, then a further high-definition image can be obtained, 
for example. 
[0050] 

[Effect of the Invention] As mentioned above, according to this invention, it is effective in being 
high gradation with sufficient S/N, and being able to obtain a high-definition digital image with a 
large dynamic range by compounding two or more images of the same photographic subject 
which changed and picturized light exposure. 

[Translation done.] 
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* NOTICES * 

JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

1 This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 

DESCRIPTION OF DRAWINGS 
[Brief Description of the Drawings] 

[Drawingjj The block diagram showing the configuration of one example of this invention 

[Drawing 2] The flow chart which shows actuation of one example 

[Drawing 3] The explanatory view showing an example of image addition processing 

[Drawing 4] The explanatory view showing other examples of image addition processing 

[Drawing 5] The explanatory view showing other examples of image addition processing 

[Drawing 6] The flow chart which shows actuation of other examples of this invention 

[Description of Notations] 

1a, 1b Solid state image pickup device 

2 A-D Conversion Means 

3 Agreement Processing Means 

4 Level Adjustment Means 

5 Image Composition Means 

[Translation done.] 
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[ISr-* (2) O^flciglSO^r^lSSn* <* 
f77'13) 0 C<D*0£. IH&t-* ( 1 ) 4Hft^- 

* (3)0«OftbOU^H*. 7OOA>6 3 0 0 OS 

[004 7] c<D£otmmmz'i7 0c±v. was 

[0 04 8 ] fcfc. fl&©S®S303<t UT, ±IE©Ii®T- 

* (1) <blj®r-* (3) ©*<D£JS*f7-*Tfcfi 

& 0 £fc. !K$r-* (2) fc (3) 

[0049] */c. «±©si»«ir«. mmmyt<D3fe 

l 1 /3ffl©^T£ttMU/c3&J 4 ffl*.fr*4te£ 1 /4{£ 
[0 0 5 0] 

l&W<D$hm) 6Lb©J:9*c. **Wtcj:titf. ^3fcS 



[■2) 
IM3) 
[04] 
30 1Mb) 
[■8] 
- h 

[tfr"t©»9i] 

la. lb HfltffifiSR* 

2 A-DSft#S 



4 u^;l/E®B#S 

5 iH&£0c#& 



1] 



[S3] 



1a 



— g 



A-D 

I 

4. 
ft 



J* 



A' 



5 



Nl 
I 



(a) 



330 512 



N2 
1 



(b) 



INt2) 



330 512 io24 



(6) 



- 1 3 5 5 9 9 



IH4) 



— t_j; 

TZX 



Nl 

I 



] r 



~T~ 



-13 



■R,6 # B«& 



-15 



ft 

(330 



ims) 

(a) 

N2| 

\ 



(b) 



200 330 



-J NO) 



200 



(a) (t>) 



N J 
1 



330 512 102 4 



N2 
t 



-INCI), 



250 512 ,024 ° 260 512 
330 330 



1024 



i 



I 



i 



11 



12 



13 



7"CMrxa3aL(2)j 15 



